This was the cri de coeur of my boss at Yale University Press.
I had been made Acquisitions Manager at Yale University Press, at least partly, on the basis of my experience as a data management contractor with McKinsey & Co. At McKinsey, I’d worked with large multinational corporations whose databases included hundreds of thousands of strings. I used Paradox, one of the most robust relational database management programs available in the mid-1990s, to break out useful data for analysis.
When I arrived at Yale in 1998, they had recently contracted with a “Developer” to build a mainframe resident database to keep track of the whole of their publishing process. SAP’s Industry Solutions for media was more than two decades away, and the custom solution had already cost YUP more than $300,000. The “Developer” was a big, combative, teary guy who’d somehow managed to learn programming. Sadly, he wasn’t very good at it. On top of that, a glitch in the wiring caused the interface to be maddeningly slow. Forget deploying a query (which the developer would have to write) to generate a report, entering a single datapoint could often take as much as three minutes.
I’d been instructed to work with the developer, figuring out how to get the expensive system to work and hopefully generate some useful information. When I asked the developer how many data rows (books that had been, or were being published) were in the database, he said “almost a thousand.” A thousand rows, no matter how many fields there might be in a row, shouldn’t have merited a mainframe solution. I would have managed that in an Excel spreadsheet. But someone very high up at YUP, someone who knew nothing about computers or data, had met with a vendor who’d promised–I don’t know what.
In a number of very uncomfortable encounters, I discovered that the “developer” didn’t know how to extract data from the mainframe, even in a CSV format. My level of stupefaction at the vendor’s incompetence was so great I could barely express my disgust to my boss. “What the hell is going on!” he demanded, again and again.
I appealed to the office “Marge,” Anne Richardson, to help me out. She gave me boxes and boxes of typewritten (MS Word) summaries of each book that she’d prepared so the data could be put into the mainframe. She’d also kept up-to-date production and sales information penciled on each title’s manila folder. When I asked about marketing budgets, she referred me to another woman in the marketing department. The design department, it turned out, also kept a set of files on money they spent on contractors. Then it was down to the basement to visit accounting. They gave me their information in Excel spreadsheets, which was very useful.
I spent a week concatenating all this data in a local Microsoft Access database. I knew how I wanted the data structured so it didn’t take long for me to simply re-key it all.
Over the weekend, I did some rudimentary number crunching that I thought would be of value. I met with the Director, the Editorial Director, and the Editor in Chief on Monday. Because I’d heard a general figure before I began my work, I asked them how many books they’d published the previous year. Their answer was off by a factor of ten. As I began to lay out all the data, pointing out fields that sold more than others, editors whose lists were remarkably thin, the relative costs of marketing or designing a given type of book, the Director’s eyes lit up. “That database was worth every penny.” Fortunately, the Editorial Director took the blow by explaining that I’d had to completely circumvent the mainframe to put these reports together.
But because of the sunk cost fallacy, the Director refused to believe his mainframe was worthless. By the time I left Yale two years later, the “developer” was still in the basement, sweating and swearing, promising things that would never come to pass.
If you or your boss has invested in hardware, software, or data management processes that just aren’t working out, don’t perpetuate the mistake–despite what the vendor is promising. Go back to the drawing board to figure out what you wanted to do in the first place, and what’s available now that can help you achieve your goals. That’s where ATC can help.