DUKE ITAC - April 1, 2004 Minutes
April 1, 2004
Members present: Ed Anapol, Pakis Bessias, John Board, Paul Conway, Brian Eder, David Ferriero, Nevin Fouts represented by Stephen Galla, Tracy Futhey via telephone, Michael Gettes, Linda Goodwin, Billy Herndon, David Jamieson-Drake represented by Bob Newlin, Kyle Johnson represented by Tim Bounds, Eileen Kuo, Roger Loyd, Melissa Mills, George Oberlander, Lynne O'Brien, Mike Pickett, Rafael Rodriguez, Molly Tamarkin, Steve Woody
Guests present: Phil Lemmons, News & Comm; Paul Corbett, Nursing; Dan McCarriar, OIT; Chris Cramer, OIT; Kathy Pfeiffer, SISS; Chris Meyer, OIT
Start time: 4:03 PM
I. Review of minutes and announcements
John Board welcomes everyone. He is playing the part of Robert Wolpert for the start of the meeting.
Paul Conway had a change to the minutes from last meeting. The entire library site is being tested with Zope, not just a portion of the site as previously recorded.
Tracy is participating via telephone.
II. Provost Peter Lange - updates and discussion
Provost Lange talks about what is happening at Duke in IT, from the executive officers perspective. He says one of the good things to think about is that we set aside money for IT initiatives and we have not spent it all on failed projects, as other institutions have done, so there is money still there for IT.
When the new president starts there will be a review of what we are doing in IT and where we are going, where we might like to go. Even before Betty Leydon came, Duke was considered an IT backwater. That is no longer the case. We have made much progress. It is good to have a historical perspective to see how far we have moved, and we have moved a lot.
We're thinking how to reorganize the academic side of computing with regard to research and researchers. Computation and life sciences is an area deserving investigation. We are looking toward the role of high-performance computing. We started the student portal. It did not cost $30 million as was anticipated two years after I started when the idea of a portal was first suggested. We have a student portal, but the evidence is not there as to whether we go forward with it. We need to figure out how much it will cost to go forward and whether it is worth it.
Melissa says we also have to consider what it will cost if we don't do it.
Tracy says this was a test. We are learning more. Much needs to be evaluated still.
Provost Lange agrees. We are all on the same page. If it keeps working, keep backing it.
How does the provost office use technology? They use e-mail at very high rate and high turnaround rate. Whenever Provost Lange has free moment, he uses his e-mail. He never leaves until all e-mail has been dealt with. He uses Blackberry as an e-mail device and telephone. It is not his only mobile phone, however, because it not safe to use in the car so he has another phone. He recently acquired an iPod and thinks it is the most wonderful thing to use when traveling long distances in a plane. He’s a heavy user of conventional technology. He uses all software programs.
ITAC should be thinking about what they want to bring into a technology agenda as it is being developed. Provost Lange would like to hear where ITAC thinks things need to be improved.
John Board asks a question that came out of a steering committee: What are Peter Lange's thoughts on the new president and where he thinks we should be going with IT?
Provost Lange says he has not spoken extensively with President Brodhead about technology. He does think the new dean of A&S has a good understanding and appreciation of technology. Duke decided some time ago that we are not going to be at the cutting edge.
John Board says we are choosing to be cutting edge on a few things, but it is largely experimental. It is cautious and introspective.
Tracy adds now we can think about selective areas where we might want to be leaders because we have a solid foundation in IT.
Molly comments that OIT is looking at ways it can expand to help departments and schools. For example, the cluster initiative, mass storage and archiving. She thinks these are good, but OIT is getting spread a little thin and things are being put aside. The LDAP is an example of a project that was moving, but then seems to have stalled.
Provost Lange says there are many issues: costs, space issues, etc.; There are a lot of roadblocks or at least speed bumps. Many initiatives will be looked at again.
John Board asks how the provost thinks OIT is doing with it’s growth and it’s changing service responsibilities.
Provost Lange says there are issues with centrally run versus individual department or school.
Tracy says it is important not to disrupt the amount of support, but to look at what areas we are supporting and where we can move resources around or combine them to realize economies of scale.
Melissa asks if there continues to be new uses of IT and real growth, and if the university continues to grow is it realistic for all the IT growth to be at the same rate. She feels current IT staff are being asked to do more and more with no addition of resources.
Provost Lange says the real question is how much slack is there or was there in the system? He feels a real advance in quality of service without significant expansion of resources. That leads to the notion that there was some slack out there. He thinks he and Dr. Tallman Trask feel Duke has been squeezing a lot of slack out of the system. Technology improves productivity but we have found often, but not always, that the support of the technology requires more resources so even if productivity goes up it does not necessarily save us money.
III. IT Security update and policy issues, By Chris Cramer
Things the Office of IT Security is currently working on are:
NetIDs – The intention is to roll out NetIDs to every person at the institution. Things like parking will be NetID enabled soon. How do we do the distribution? We do it for students, but it is not really secure. We are using now the same system, same paper and printing, as we now do for paychecks. We are testing a challenge response system. Long-term we would probably like to make it an online system. There are issues there, but we are making progress.
Password cracking – IT security runs a password cracker quarterly. If we crack a password we send an e-mail and tell someone to change their password. If they don't, we send another. How many times do we do that? The intention was to do it 3 times and after that we lock down their NetID. What does ITAC think we should do about people who do not change their password after three warnings?
John Board thinks 3 months is too long.
Paul Conway says UNC requires passwords be changed every 90 days.
Chris says this is different. We are only telling people who have weak passwords to change them. If you have a strong password, we won’t issue a warning.
Raphael does not understand if the password is cracked, why don’t we say you must change the password next time you log on.
Michael Gettes says it is a hard thing to do because given how authentication works we couldn’t notify all applications that the password change flag was activated for a user.
Chris says we are not saying we will lock someone out if they change their password to another weak password, we will lock them out if they don’t make and effort to change it at all. Chris’s problem is that he is ready to go live now, but what does ITAC want him to do?
Bob Newlin says the Med Center already has a good system for changing passwords. He thinks Chris should tighten his timeframe to shorter than 3 months.
John senses ITAC is in support of this and in support of locking the NetID as the punishment.
Tracy says before we go forward we need to have a strong education campaign out there first.
Someone suggests the lock downs and warnings are staggered to take some load off the help desk. All nod in agreement.
IV. SISS Senior registration, By Chris Meyer and Kathy Pfeiffer
A few minutes before the senior registration window opened the server started thrashing because of increased load on the memory. Oracle and IBM were being consulted. The net effect was a reduction in the amount of memory. Sunday afternoon about 2 GB of memory leaked. Memory was reallocated and all processes on the database server were shut down and restarted. Registration was successful. Registration took place every morning this week and was monitored closely.
Kathy Pfeiffer says over the years we have gotten better at crisis management. So many hands on deck and so many new tools were available to monitor things and we were able to shut down registration very early in the process. And we decided to take our time in trouble-shooting rather than start back up after a few hours. We need to think about how students are registering. We have new technology, but we don't know when to expect the load, should we space it out more? Given that we had two major upgrades, one in Oracle and one in Peoplesoft, this was not considered catastrophic.
Brian Eder asks if we have looked at other institutions to compare what they have done with similar equipment and tools.
Chris M says we can't really compare because no one else is running what we are running. What we are running is custom. We don’t have the hardware now to test like we did before.
Tracy comments that most institutions are not registering one thousand students in 3 minutes as Duke is.
V. Email system update, Michael Gettes
We are in the middle of a migration. We are two thirds through. Two thirds of users are on the new machines. We received an error notice that there was a corruption of the file system. No mail was lost to anyone. We had to rebuild disks. Basically we put things back together.
Then on Thursday we began to see more file system corruption error notices. We contacted Sun and the problem was given the highest priority. About 20,000 messages were delayed in delivery. Duplicate hardware was set up and only today have we been able to get it running.
We are concerned about the remaining file system. We have not seen more file system errors, but we are not comfortable. We are scheduling a downtime tomorrow evening for 12 hours. This means we will have another downtime afterward.
The cooperation the Sun has given us has been terrific. We discovered an OS problem too. About two weeks ago, Sun discovered an error in Solaris. We have multiple compounding problems. Sun provided a software patch that will be installed tomorrow during the downtime.
Tracy wants ITAC to understand even though the disk arrays failed, it is not a simple matter of replacing them with another hardware vendor’s products. That would bring in changes to the whole mail environment.
George Oberlander asks if all the drives in the arrays are from the same manufacturer.
Michael says they are the same on the old system and there are issues there, not so much with the drives but with the controllers. Drives fail, but controllers should be able to handle it.
John Board recaps what will happen: 12 hour downtime Friday night. Saturday mail will come up on the duplicate array.
Tracy adds, as worrisome as any situation like this is, she is more comforted by the fact that not only do we have a full list of resources from Sun, but also Michael and team are being very methodical in their diagnosis. She thinks we will get there with not so much worry.
John says in a meeting yesterday everyone was amazed that it happened to us. We are small fish in Sun's pool. This would be expected somewhere like Amazon.
VI. Other business
Stop: 5:24 PM