Thursday, October 11, 2007

Caspio dustup

Some of the latest chatter on various blogs and boards is about Caspio, which bills itself as an easy way to get your database output to the 'Net. (OK, that's simplistic, but it'll do for now.)

That's touched off a wider round of posts dealing with database journalism and how it's lagging in the eyes of its beholders and (sometimes not so subtle subtext) how news organizations are too cheap to hire the journalist/programmers needed to make it work.

Adrian Holovaty, of course, juiced this discussion way back in 2006 when he proposed that all newsrooms needed to stop thinking of story as a river of text and start thinking of it as creating and intertwining databases of the information they gathered (more on that from an interview he did with Online Journalism Review).

Lately, Derek Wills of the Washington Post has taken a couple of swipes at Caspio. And as you can see from the comments in that first post, the system has lots of defenders (I noted particularly Peter Zicari's comment. I know Peter, who works for the Plain Dealer, from the American Copy Editors Society and put a fair amount of trust in what he says).

Jacob Kaplan-Moss fleshes out the debate a little more with his recent post.

Looking at this with a programming background that goes back to Fortran 4 and my days in astrophysics (don't ask) ... but whose skills have rusted with lack of use (rough translation: I understand the lingo, know the questions and issues, and couldn't write current code if my life depended on it) ... and who knows the journalism end of it, some thoughts:
  • Yep, Derek & Co. are right. We should be training database journalist programmers
  • We should have development tools that enable those DJPs to do natively what can be done as quickly as with Caspio
  • Ain't gonna happen soon
Read Zicari's comment again, and you'll find a lot of wisdom in there. And that's a "large" newsroom. It's always a balance between speed, ease and completeness/flexibility. And all of that balanced against cost. Right now, so much of this industry is searching for something, anything, that the cost factor (as Jacob outlines it), I think, pales in comparison for executives. They want it quick and, if necessary, slightly dirty. It's about eyeballs, pure and simple, and if two tin cans and a string would attract a crowd, well, guess what ...

Now, to the two other parts of the debate: 1) We need more programmer/journalists and 2) we need easier tools.

As to 1: With due respect to Matt Waite, Adrian, Derek, Jacob and dozens of others I deeply respect, it isn't likely to happen soon. In many cases (and let me stress, not all) the university system isn't really equipped to handle this. First, it requires cross-department and, perhaps, cross-school cooperation. That's a mouthful alone if you know anything about the way most schools operate. And then it can take from one to three years to get a course approved (one does not just create courses in most schools; there are curriculum committees and university committees and other departments that must weigh in).

(For now, let's not get off on a debate about whether higher ed is set up to deal with the new world order -- let's just stipulate.)

There are other structural challenges, too. Our school, for instance, has a "visual communications" track as well as the traditional "journalism," broadcast, ad and PR tracks. Right now, viscom would seem the natural place to put something like this, but it pretty much ends with Dreamweaver. Viscom is annoyed at broadcast because broadcast's curriculum is so tight that it doesn't have spaces for viscom students who want those skills (so if it's this difficult within a department ... and we're not the first. Go research Kansas, for instance, or BYU). We've been struggling with how to fit this all in (if you want to see the report and a pretty good compilation of resources and links, go to our wiki).

Second, accreditation standards add to the complexity. Accredited journalism programs are under some fairly tight strictures as to how many hours can be taken in the major -- to put something new in, something generally has to be bumped out at the same time we're also getting complaints that students don't get enough writing and reporting (again, stipulated). And, sure, you can say ditch the accreditation. Certainly some large and well-known programs don't have it. But that again ignores reality in most schools whose faculty depend on faculty from other "accredited" departments to grant promotion and tenure (disclosure: I am a lowly untenured instructor, so I get to watch the fun and games from the cheap seats).

Third, university finances and financial systems pose challenges. Pump "value centered management" into Google and start reading. (Or you can get a broad range of perspectives here.) Now, certainly not every campus is using VCM, but there plenty of variations of it floating around and the bottom line is, in general, that they discourage cross-department "sharing" of students unless the revenue you lose is offset by the revenue you project to gain by bringing in students from outside your department. As anyone who's ever done financial projections for business knows, that can be dicey.

Fourth, there's the idea that many state schools, with declining state support, are focusing their efforts on areas that can get federal grants (because those grants pay substantial overhead) or major private foundation mney. So far, Knight has provided some juice in this area, but let me suggest that even at its level of total funding, the response from most of those in the physical sciences would be "that's nice; I think we have that over here in loose change." Is the industry willing to step up and fund this kind of research? (Hey, I love rhetorical questions.)

The second part of the challenge -- developing tools that allow journalists to do this well, and on their own servers -- may be the more achievable. And I would urge those who have the skills to consider it. But when was the last time you saw this industry cooperate on any kind of software (or much else, for that matter)? (That could take awhile, so let's move on.) I've been urging press associations recently to take up the challenge using the cooperative model. Much as they have formed advertising cooperatives, I think it's time they considered IT cooperatives to share the cost of such development, especially for smaller papers.

It's clear from the exchanges over Caspio that the old vein of journalistic independence (why would you want to put your data on their servers) remains strong, even in the digital age. But then again, isn't that the same mentality that said you had to own your printing press, which is how, in my opinion, lots of news organizations got into trouble to start with?

If nothing else, once you strip the tech talk away, all the words written about Caspio once again highlight the deep and difficult issues this industry (and the schools that feed it) faces.

Labels: , ,


At 10/11/07, 10:59 AM, Anonymous Anonymous said...

I know firsthand what it takes to develop web apps. I was the senior manager of the Internet Technologies Group at The Atlanta Journal-Constitution prior to joining Caspio. My experience has been there are many web apps being built that short-cut the process, including having approved documentation, design, coding, and quality assurance standards.

Not to mention having the web systems administrators who can account for server performance, load-balancing, security, etc.

And where’s the project and product managers in all of this?

Pile on top of this the cost to have multiple virtual servers so you can have machines for development, testing, deployment and failover.

Now the cost of doing business has gotten a tad bit more expensive than some journalists are willing to estimate.

I know the AJC has all of this in place because I helped build the infrastructure that is 100 percent LAMP-based. That said, the AJC continues to be a satisfied client of our Caspio Bridge product.

This is why so many media clients use Caspio Bridge. It is a wizard-driven framework that doesn't require programming skills. But Caspio is also used by senior level developers at media companies because it allows them to cut their development in half.

There's also a lot of misinformation in some of the posts you link to concerning our product and how it is priced. So please allow me to set the record straight.

Caspio charges a monthly fee based on the number of DataPages in your account. DataPages are what we refer to as database-driven applications that are served by Caspio, but displayed seamlessly on your website. A DataPages can be used either standalone or together to make end-to-end application. Examples of a single DataPage include a web form for new survey submissions; a multi-screen interface for search, details and data drill-down; a dynamic web-based report; or a password recovery system that retrieves lost passwords. We charge between $7 and $12 per DataPage per month with no long-term agreements, no contracts to sign, and the ability to upgrade, downgrade or cancel the account at any time.

Included in that price is …

- Powerful and secure database platform;

- Un-metered tables, records and submissions. Gives you flexibility and freedom to scale your applications affordably;

- Free support and project consultation. We personally help you launch your app;

- Active performance tuning to optimize performance in large or complex apps;

- Up to three web-based training classes (beginner, intermediate, advanced) for 10 staff members per site per year;

- Free Secure Sockets Layer (SSL) for 100% data encryption;

- A customized Cascading Style Sheet (CSS) to match your sites' look and feel;

- Up to 10 free “sandbox” DataPages per sub-account, which you can use for experimentation, testing and future apps;

- Unlimited app users and no per-seat fees! Our pricing is based on the number of applications – not the number of end users;

- Daily data and applications backup (archives are retained for 14 days);

- Free project consultations via the phone or web with a Caspio Solution Advisor;

- Automatic product upgrades;

- Free ready-made apps. Deploy "as-is" or customize as needed;

- Free newsworthy eAlerts when the federal government releases major data sets;

- Free scripts to do Yahoo! and Google mashups;

- Advanced application integration. Caspio's Relational DataPages, Web Services, and Plug-in for Microsoft Office let you enhance your app and interact with other applications. Our free SOAP-compatible Web Services enable developers to enhance Caspio Bridge applications with custom functionality. Web Services can be accessed through virtually any programming or scripting language including ASP, .Net, VB, PHP, C#, Cold Fusion, Java etc. And the free plug-in for Microsoft Office that automatically downloads your Caspio Bridge data to your local desktop and keeps it refreshed.

And yes, Caspio Bridge is available for in-house installation. We will provide everything the customer needs for installation in virtual server(s) or assist them to install it on their physical servers. The cost, which includes support and upkeep, is a monthly fee similar to Caspio Bridge Enterprise service at 500 DataPage or higher capacity plus a setup fee of $2000. A one year commitment is necessary. Details of the package are at

Caspio gladly permits anyone a free 14-day trial who is interested in evaluating the product for use. A trial can be requested at

Should you have any questions, please do not hesitate to contact me directly at (650) 691-0900 x741.

David Milliron
Director of Media Services
Caspio, Inc.

At 10/12/07, 10:34 AM, Blogger Derek Willis said...


If universities can't or won't be part of the solution, that's on them. I took a single computer class in college, and that was in 1989, so I don't imagine that it was integral to my development. I would venture the same guess for Matt, a journalism major.

So if we also can stipulate that people can (or must) learn outside of the classroom, then my original points on Caspio are relevant, imho. That's because they represent the difference between handing out fish or teaching someone to fish.

While I sincerely believe that there are many people who are utterly happy to use Caspio and similar products, I also know from personal experience that you learn by doing, not by having things done for you.

If you think that database journalism is important, then you probably would want the people doing it to broaden their expertise and increase their capabilities. There are situations where tin cans and string, as you say, are just fine. I just don't think that's a good long-term policy for the industry, and I'm curious as to how much time you think we actually have to put up tin cans before other people do this stuff better than we do now.

Derek Willis

At 10/12/07, 9:10 PM, Blogger Doug said...

David: Thank you for the details. They add to the discussion and I hope will be viewed and used by others.

Derek: I trust you detected that one of the central points of my post was not to cry on anyone's shoulder but to express some frustration about why jounalism education may be at risk and why the economic reality is such that services like this would get a favorable look.

Everyone has made cogent arguments: short-term cost vs. long-term benefits, skills development plotted against time to project completion, etc. Your position is just going to be a very tough sell inside a lot of organizations -- as str all such things whose benefits are long term and difficult to measure. Long-term you have the better position -- the more data-trained journalists, the better (also interview-trained, research-trained, storytelling-trained,common sense-trained and crosss-media trained).

But journalism programs largely exist in institutions not set up to turn quickly, as we in this busines need to do. Those institutions also demand that the constituencies behind such professional programs pay to play (can't put it any more simply than that). But the journalism community, while happy to take the cannnon fodder that such schools produce, largely ignores, or purposefully remains ignorant, of the realities.

I have feet in both camps: 30+ years in the business and six in the classroom (eight if you count an earlier shorter stint as a Kiplinger Fellow). And I still "do" journalism as time allows.
I think your posts and some of the others I link to go far beyond the debate about a database service and serve to illuminate some of the bigger issues, which is why I jumped in and the angle I came from. We don't often, in all the debate on j-education, hear the gritty nuts and bolts from the other side of the ivy-covered walls, so I thought I'd throw some of those in, too, to enlighten the debate. (Here's a challenge, BTW, to everyone. Next time you write some story on some big multimillion-dollar research grant, ask what the "overhead" is and how much actually will go directly to the research and researchers. Now, be fair, overhead is needed to keep the lights on and the heat and air conditioning running, and the university administration paid, for instance. But just ask anyhow and make your own judgments -- in other words, do some journalism.)

Perhaps a serious contraction in journalism programs is needed. I suspect we will have fewer and fewer j-programs turning out people truly equipped to deal with the modern media world. Many of the rest will become "mass communication" programs, in fact, if not in name. There is a stark difference (and a bit of an irony - yes? -- in an era when communication is becoming far from "mass").

At 10/13/07, 2:43 PM, Blogger Derek Willis said...


My position is going to be a tough sell as long as the industry accepts the pitches of Caspio about the economic and other costs of database journalism as gospel.

To be clear: businesses have an absolute right to do business and make money, and Caspio has done well to recognize a need in the marketplace and try to fill it. But let's not forget that they have a clear economic interest in discouraging their clients from moving from their platform and hiring people to do this kind of work. They have every reason to tell you that database development is too hard for journalists.

Your "economic reality" of database journalism seems to be at odds with the example of the newspaper that employs Jacob Kaplan-Moss. Hiring developers has enabled them to actually hire *more* developers and do more work. And at a relatively small newspaper, to boot.

In that case, and in others, the benefits are not, as you say, "difficult to measure." In the same way, some of the features of Caspio apps, such as a lack of findability by search engines, also are measurable.

Let's agree that economic conditions are tough for the news industry. But please consider the impact of promoting a "tin can" model when other options - some of them quite affordable - are available. If this discussion is important one, then it deserves a more thorough and balanced treatment.


At 10/14/07, 3:39 PM, Blogger Doug said...


I get the impression you think I am on opposite sides from you -- far from it. I believe this industry -- and individual newsrooms -- need to develop their own databases and software.

I repeatedly have decried, for instance, cookie-cutter cit-j sites such as Your Hub because I think that to be successful, you need to caputure the quirks and flavor of your community (real or virtual). I think the proprietary publishing system software vendors -- I won't mention names; we all know who they are -- have been some of the biggest impediments to innovation in this industry. Their proprietary systems were hardly blog friendly until recently, for instance, and I still hear of instances where they will not open the door to their APIs so that publishers can buy third-party apps that are far ahead (think classifieds, for one).

Yet, I also work with some smaller papers, I have just finished several seminars for press associations, and I can tell you that economically in the short and medium terms (and let's not get into the long term vs. short term debate -- it takes a helluva visionary to stick to the long term) those systems make economic sense for them. It's maddening.

What I'm trying to do is put your clarion call up against the map of economics and suggesting it's not going to happen any time soon if we don't dissect the economic disincentives and deal with them. Appeals otherwise tend to fall on deaf ears. Right now, the economic incentives are not there, neither in the business nor in the academy.
Jacob's paper is not the typical model; aside from a visionary management, it sits in an unusual college market and it has an economic concentration in that market that exceeds that of most media operations.

I have been urging press associations to consider a cooperative model for software development much as they now have their ad cooperatives. Many smaller and even midsize papers will not have the horsepower to develop useful apps (or customize those out there) without the ability to share costs. Many do not have the ability to host their own sites (which is why you see so many with other vendors, and I've seldom seen one of those sites that I liked).
Companies like Caspio see those economic dislocations and, of course, fill the void.

Perhaps we need to find a way to join force to set up these cooperatives. While it wouldn't bring such things in house entirely, it might be closer to what you envision.


Post a Comment

<< Home