Tell us a bit about your work.
For twenty-six years, I was with Central Management Services (CMS) for the State of Illinois. CMS is like a super agency that provides services to other state agencies. They run the payroll system and do the health insurance and other benefits for most everybody that’s employed by the state, including universities. There are some large agencies that do their own payroll.
I used to work on the application side developing for mainframes before I moved over to storage. On the storage side, I worked with state agencies to make sure they had enough space on the mainframe to hold their files.
What did the storage side entail?
We managed equipment for state agencies and billed for CPU, disk space, and all that. If equipment was shared among a few of our agencies, we only billed each agency for what they were actually using. But there were some storage areas that were limited to one agency—since they were the only ones that used it, they got billed for all the storage, whether or not it was being used.
We set a notification level of 10 percent. So if you had less than 10 percent storage capacity remaining, you got notified to add more. But, you know, the agencies didn’t want to pay for another storage volume. We’d call them up and tell them they needed more, but they’d say, “Let it drop to 9 percent, 8 percent, 7 percent—we’re still working!—5 percent! Four percent!”
Eventually, they’d say ”We’re not working any more. We need more space.”
Why do you think they cut it so close?
They didn’t want to pay for it. If we added a volume, which was eight gigabytes, they’d have to pay for that space even if they didn’t actually need it for a file. They didn’t want to drive up the computer budgets.
But when you’re operating like that, any tiny little bump can stop your work from happening. And it did from time to time. One time, this agency had a log get up to the maximum number of extents. The log itself wasn’t too big; it was just using too many little chunks of memory. They called me and said, “We can’t do this transaction!” Everything came to a screeching halt and their system shut down until they had defragmented that log so that they could gather more space for it.
That sounds like what’s happening to these government systems that are crashing as millions of people file unemployment claims. What’s your perspective on all the recent news about that, and particularly the criticisms of COBOL? Do you feel the criticisms are warranted?
Totally unwarranted. COBOL is the backbone behind the businesses on the web. Your credit cards would not work without COBOL. American Express, Visa, MasterCard—they’re all running COBOL. Airline tickets? The backbone behind that is all on mainframes. The airline industry developed software in the 50s on specifically designed computers. The software they run on mainframes today is directly descended from COBOL code they wrote for the IBM 360, which came out in the mid-1960s.
COBOL was designed for accounting. It’s designed to be exact to the penny. The IBM 360 came out with instructions that fit into that kind of paradigm. When you go to any other language, they don’t often have decimal math instructions for that kind of accounting work. So any time you port COBOL over to another language, you can get your basic logic over and then you’ve got to test your accounting and make sure you come up to the penny again. There are often rounding errors that creep in after a while: you start losing your least significant digits and, eventually, you’re no longer accurate to the penny. That does not happen with mainframes running COBOL.
COBOL’s strengths aside, these systems are struggling. Do you have any thoughts based on your experience about why they’re struggling?
None of the other recessions we’ve had since 1918 have come all at once like this. You start to have a recession and your unemployment systems are handling 0.1 percent more, 0.2 percent more. Your systems have time to grow and scale. You’ve got more workload handling the new people coming in while paying everybody else out from their last week’s records.
Today, you’ve got 10 percent of the US population laid off and filing for unemployment within months. That’s one hundred times your normal recession workload. The employees who process those claims are completely overwhelmed. The CPU processing speed is overwhelmed. Your disk storage for all these new people is completely overwhelmed. If you’re going to 15 percent unemployment and you were at 5 percent, you need three times as much storage space. And the people who work in the unemployment office are trying to handle fifty to a hundred times their normal workload. You can’t just enter your information and automatically get unemployment. A person has to sit there and approve or disapprove every request.
Yeah, we work a lot with government caseworkers and there’s always a human that has to do some sort of thing on the other side of the screen. When I talk with friends who work at Facebook or wherever, they think it’s this perfectly oiled mechanical system. But there are humans all over it, and they do really complex things. And humans are hard to scale.
That’s right. At CMS, we worked with the agency in charge of unemployment. The only person we ever saw from that office was a shop steward who would come down for meetings. She had a guy transfer from the Department of Corrections who had trained to work on the unemployment system. Of course, it had lots of complex rules and you have to have them all memorized. It takes weeks, if not months of training to learn all the rules you have to follow before you can say yes or no. So this guy spent months in training, but in the end, he found it so stressful to be handling the unemployment claims that he went back to corrections.
How do you actually sustain that when the software is so complicated? And we’ve seen several times in our government work that one or two people maintain ten systems. What happens when they leave?
The complexity of these systems is a big issue. On the technical side, if there’s only one person who knows a system and that last person goes to retire, there’s nobody left to train new people on the software. And the mainframe software is so much more complex to use. With Linux or Windows applications, you get trained up in a week. But for mainframe applications, every installation is custom. There are tricks of the trade that new hires have to learn by osmosis, by working with the prior person. A new guy comes in, you tell him what to do, and you sit there and hold his hand while he goes through and does it. And you are still learning new tricks even after a couple of years.
Now, if you’re the last person and you’re not training anyone, all that knowledge goes away. A new person who comes in and starts working on a system that hasn’t been maintained for three or four months. People try to write everything down but you never say a lot of these things, even to a person sitting right there next to you.
If you want to see somebody actually working on the mainframe screens to get a sense of how complicated they are, moshix—that’s his username—on YouTube has about 160 video recordings up.
Have you watched some of these?
Quite a few, yes. Some of them I watch because I’ve never used the products he’s working on. Others remind me of systems I’ve worked on and I find it enjoyable to watch it over again.
Get Your CICS
You mentioned earlier that you were on the applications side at CMS before you were on the storage side. Can you tell us about when you started and what it was like?
When they hired me, they were moving into a brand new office building. There were about 150 people there, and we had thirty or forty mainframe applications. We had CICS, IMS DB, just about everything. We were a service bureau.
Teams now tend to have product managers, designers, and programmers. What was the makeup of your team?
We had systems analysts and we had coders. I was in-between. In terms of designing the applications, you’d start with a screen layout and a database layout, and go from there. Usually, you’ve got a form that you need someone to fill out. You have to replicate the form fields on screen: first name, last name, middle name, social security number, address, and all that. That gives you all your records.
The first IBM was this Hollerith card system they used for the 1890 census. You’d have a person’s name, their address, and all this other information punched on cards. Of course, it wouldn’t all fit, so you’d need a few cards for each person and you needed to know what information was on what card—card one had the name and address, card two had the spouse and job, card five had whatever. That’s your set of information for that person.
Now that you’ve got computer files on hard drives, it’s all in one record. All the transactions for that person—having a kid, changing their healthcare plan—those are different fields under that person.
It sounds like, when you were building out features, there was programming you did on the backend and then programming in CICS. Is that more like the frontend?
CICS itself is a multitasking transaction manager, where it takes in a transaction name that the user enters. It uses that transaction name to start the COBOL programs associated with it. A transaction in this case is like a thread. Maybe CICS runs the programs, then returns the result and exits. Or maybe it shows you a result on the screen, asks for more inputs, then does more work, and then exits. So CICS itself would not change but you’d get releases every other year with updates through the operating system.
When you found a problem with your application program, you’d put the changes in at that point. Only a few people could update the authorized production application, because you just don’t want a bunch of people going in and messing up a production database, causing all kinds of problems. The real problems were bad enough.
Did you all do any user testing where you’d have a caseworker or an HR worker go through that screen with you?
Yes, we’d call up our end users and they’d try out the changes to the system themselves. Everybody who did the testing had a coax switch to switch between test and production.
Like a physical switch?
A physical switch on a coax line, yes.
Software engineers now expect to be on call and sometimes get called in the middle of the night. Were you on call for these systems?
Yes, I was. All thirty-some years.
We would have two or three people in a group, so we could rotate. But then we got people in that had grown up around PCs, and they just don’t appreciate that computers have to operate 24/7.
How did being on call for thirty years affect your life?
Since I did it my whole life, it’s hard to say.
I was limited in how far away I could go for a social event since I had to be able to get back to work if called. And limited on drinking since I had to be able to drive to work at any time.
Most nights I didn’t get called, but then some nights would be bad and I’d get two to three calls in one night. Other times, I wouldn’t be called but once a month. When I got towards the very end of my career, it was less than once a month because we kept so on top of those storage groups, made sure we had enough space on them.
When you were working on those applications and you were working with people who then were using your application, how did that feel?
I felt a real pride and ownership in working on that screen. There were 300 to 400 people who used it to sign up 80,000 state employees for healthcare programs all across the state. It was important that it worked just right for them.
These people are all counting on me to do my thing right. I felt quite a bit of responsibility.
Make and model
Backing up a bit in the story—how did you first get into computers?
I was in high school in 1979 and they got in two TRS-80 Model Is with a cassette tape drive and sixteen kilobytes of memory. We started doing little projects on that. They had one in the electronics lab and it was empty during my lunch hour, so I would go down there and use it then. That got me interested in computers.
You could sit there and see your results right in front of you. I had even used a teletype printer keyboard at this physics camp at the University of Wisconsin-Whitewater in June 1978. You type in a command and it’s printing as you go. Then, when you press enter, the command goes in and starts printing up the results.
I got my computer science degree at Illinois State University in Normal, Illinois. I used punch cards for my first two classes in the fall of 1981 and spring of 1982. We got IBM 3270 consoles at that time, and that was also when the IBM PC came out. I also had one class on an Apple II computer with a Zilog Z80 card in it, which was a follow-up to the Intel 8080.
Could you describe the teletype printer keyboard more? Was there a screen?
We used it to play this Star Trek game. You didn’t have a screen; it was all printed text. You’d type your command at the printer because it was a teletype interface over an acoustic coupler modem that had cups to hold the phone headpiece with speaker and microphone. The teletype machine would output sounds that the cups would then receive and transmit through the modem. The telephone system didn’t have the RJ45 phone plugs to directly connect phones into the modem yet.
When you were going to college at Illinois State, why did you choose computer science?
The IBM PC didn’t come out until November ’81 and they didn’t have one at the university until fall of ’82. At that time, if you wanted to use a computer, you had to work for a big business that could afford to buy one. Even the TRS-80 was a couple thousand bucks and I was earning three bucks an hour. I was not going to be able to get one. I knew a computer-oriented job would be with a big company that had an IBM mainframe or a minicomputer.
In 1981, the IBM 360 had just been out fifteen years and they were still rolling out major changes to the operating system. It had thirty-two bit registers but only used twenty-four bits for the address so it could only address sixteen megabytes. Our mainframe computer at the university had eight megabytes and when they expanded it to twelve, it took a whole refrigerator-sized unit to add that amount of memory. Every four megabytes basically meant another refrigerator. And then the computer core itself was one or two refrigerators.
So where did you end up working after college?
I applied to the Illinois State Police (ISP) because they had a job posting. I went and interviewed, and they initially said they were going to hire me, but then they called back and said I wasn’t going to get in because I had voted in the Democratic primary. They told me to contact my county Republican chairman and join their Young Republican’s group, which I did. After that, ISP offered me a job. I started on December 3,1984 and I worked on various applications there. One was crime statistics reports, and then I went to online applications for them. Never involved in cases; I was just making sure the computer systems were up and running to track traffic tickets and, later, investigatory files.
What were your first impressions of that job?
The office building was sixty years old by the time I got there. You had a desk on one side and a computer table on the other. There was a 3178 display, which is basically a CRT unit and a keyboard with a little computer box on the side of the computer that communicated over coax cable to the box in the closet. There wasn’t a full PC there yet. It worked really well. In fact, I kind of missed it when we got our PCs. The PCs wanted everything in twenty-four lines instead of forty-three, so we had half the amount of screen we’d had before, but it seemed like even less. But other than that, you know, it’s all the same information.
Could you talk about the applications that you worked on while you worked there and who used them?
It was the dispatchers at the state police headquarters who talked to the troopers over the radio. Entering traffic tickets, entering a license plate to do a query on it. Find out who the owner was, if they had any warrants out on them.
That system had been around over a decade by the time I got there in ’84. It was about ’69 or ’70 when they got their first computer going to communicate with the FBI and offices in other states.
It was a federal project?
There was a state project called LEADS that communicated with the FBI’s computer and then each state had their own computer and applications, just like every state has their own state police.
These were applications written in COBOL, Assembler, and IDEAL. Then I did a reporting system on a CICS with Datacom. For a drug traffic stop system, I copied an application we had to a new application called Valkyrie. It was to track what they found during traffic stops: drugs, weapons, whatever was found. I think we were the first state to come up with an application to do something like that, and that was one of the last projects I got to do over at the state police about 1989. The headquarters for that was going to be in El Paso.
We work on technology to help clear people’s criminal records. So I’ve talked to a number of people at the FBI and Nlets about making sense of criminal record data across state lines.
Yes, it’s complicated. The FBI has their own network for sending electronic law enforcement data nationally, but the states can’t use it. The states had to create their own organization called Nlets to communicate criminal justice information, like out-of-state arrests, across state lines.
There are other barriers too. You know the 1-800 number you have to call to do the background check for a firearms purchase? By US law, all that information has to be stored on paper. So when you call that number, somebody takes the person’s information and goes and looks it up on paper and then calls you back with the information. You can’t just type it in and get the information back, or even just have a person in FBI headquarters type the name in and find out right there. The FBI person doing the lookup has to go physically search through paper for that person.
And Congress doesn’t want the information released. So they just put up all these obstacles in people’s ways.
Why did you leave ISP?
Well, that was the asbestos. We were up on the sixth floor and we had like skylights up there and the caulk would leak from time to time. Sometimes, we’d have a leak and you’d smell the mold. And then that started dropping on the asbestos and it all started falling off the pipes and onto people’s desks.
One person had a four-inch-high conical pile of white powder on their desk every day for about two weeks. We thought the janitor was pulling a prank or something because they’d sweep it off in the morning and there’d be nothing there all day. Then we’d come in the next day and there would be another pile there.
When we found out it was asbestos, we were concerned. We reported it to building maintenance and they covered it with duct tape and started taking air samples. This air sample machine was sitting there every day making noise. We lived with it. They moved us to another floor while they fixed it up, and when they were done, we went back up there.
You went back?
Yup. But that was the last straw for me. Repairs were going so slow. They were there for another twenty-five years in that building. It was 2009 before they bought the Franklin Life buildings in town and got the people out of there. It was 2012 before they got the computers out. The rest of the building still has to be cleared out and they haven’t done that. It’s been sitting empty for close to a decade now.
Union Dilution
Were you covered by the union at that time? And was the union present in those conversations about the asbestos?
Most of my time at state police, I was in a merit comp, a nonunion position. The “programmer analysts” were in the union, but we were just “programmers” so we weren’t. Right before I left, around ’88, they brought us in.
Do you know why they did that?
When the union wants to bring in a particular title, they send out letters to everybody in that position. When the union gets back so many letters, they can go to the National Labor Relations Board and call for an election where everyone votes on whether they want union representation. Enough of us voted to join.
By the time you moved to CMS, were you always in the union going forward?
After that point, yes. I told ISP in spring of 1990 that I was going to go over to CMS. The US Supreme Court was sitting on a case called Rutan versus Republican Party of Illinois over the hiring practices by the state of Illinois. There was a transfer freeze and hiring freeze during that whole period, so I sat there for six months working on a project not knowing when my transfer would go through. Once that decision came down, all the transfers went through all at once.
What union were you a part of?
AFSCME. Different locals represent different agencies. There was one for the police officers in the state police department and then there was another for several agencies that have employees in Springfield, just the employees in Springfield. There was Local 2224 that covered CMS and five other agencies in Sangamon county. And then Council 31 has all the other state employees in different locals. So they were running the show whenever they were negotiating contracts. In about 1994 or so, I actually stood election for negotiator and won, so I was involved in those negotiations for about five months. We’d go in for a Tuesday, Wednesday, Thursday negotiation session. And then three weeks later we’d have the next one. Each local could send five delegates, so we had something like 250 people total for our side and then there’d be maybe a dozen agency heads for the management. We’d meet in convention centers.
What else would the union work on?
For example, my coworker was applying for other positions and she had the seniority and the skills to take these other positions. But she would never hear back about her applications. Most people, when they don’t hear anything, they just assume that they didn’t get it. But she would follow up, find out who got the position, and then compare their record to hers. If she had seniority on them, she’d file a grievance. She’d get the raise she should have gotten before, though it wasn’t backdated to make up for all the time she had been underpaid. She was stuck and trying to go someplace else for fifteen years. I know that happened to her at least three times. And it just seemed so cheap, you know. Her seniority should have applied—your seniority is supposed to give you priority within your bargaining unit, within that county—but the state wanted somebody else and it didn’t cost them much.
The state basically just hires whoever they want. And if the person who they should have hired eventually found out, then and only then would the state have to give them a small raise to match what should have already been their pay. It’s a low-cost violation for them.
When did you first experience contractors in your government work?
When I was first hired on at the state police, they already had some contractors and we basically treated them the same. The main thing was, on the phone list, they all had an asterisk by their names. And if they did something wrong, then boom, they were gone and there was nothing they could do about it.
The big thing about being a state employee is you get a pension and health insurance. The state has to do that to keep people there, and if you as an employee want to keep those things, you have to stay with the state of Illinois. But if you were a contractor, you didn’t get anything else. They were paid twice what we were, but no pension or healthcare. If they found better wages or something more interesting somewhere else, they could be gone in two weeks. Lots of people stuck around, but they also might be gone in a year or two when their project wrapped up.
What sorts of roles did contractors fill?
Application programming for the most part. The union had put into the contracts that agencies couldn’t do a personal services contract. So that meant that they could never interview an individual contractor. The contractor would have to incorporate into their own company or be hired through some other company. But they were basically just employees.
We got rid of those for the most part about 2005. The union got the state to recognize what state agencies were doing. Now, if you want an employee, you’ve got to hire an employee. You can’t just hire a contractor through a third party and say it’s not personal services. The state had a year to hire replacements for those people. That was quite a bit of turmoil: a lot of people to hire and a lot of expertise was lost. I mean, some of the contractors had been in there for decades.
Did it seem like that was one for one, like one employee was hired to replace every contractor, or were there some contractors who were never replaced?
There was considerable shrinkage. It’s worth noting that by this time, the state of Illinois had cut taxes, yet kept up their expenditures, so they were behind on paying bills and falling further behind every year. They were borrowing out of our pension funds just trying to catch up on bills. Today, Illinois has something like five billion in unpaid bills with about six months delay.
Back to your question about contractors specifically: the agencies tried hiring, but they didn’t get many replacements. Many contractors didn’t apply for their own jobs as employees because the pay was lower. I saw the job postings go up on the bulletin boards. Normally, there would be enough surface area on the board for each listing to have its own place. The listings were so thick you had to leaf through pages and pages. Even after a year, they had all these vacancies they were still trying to fill.