SAPbiz - January 13, 2009

2009 Support Pack Update Project - Lessons learned

Participants should brainstorm everything they felt was successful during the project life-cycle (the "Wins") as well as everything they felt could have been done more efficiently or effectively (the "Challenges") including a recommendation for remediation. This can include but should not be limited to Communications, Governance, Process, Project Mgmt, Risk Mgmt, Sponsorship, Staffing Levels, and Teamwork.

Communication - Tools that included Meetings, Emails, SAP Gui & Web messages, Conference bridge line, Status check-in line and the Project Wiki were utilized to communicate project status, roles and responsibilities and system and service outages.

Was the appropriate volume/frequency of communication provided?

Was the information accurate and complete?

What worked well? Where did you see opportunities for improvement?

Wins - Communication

• WIKI good. Used it. Good to have meeting notes.
• No negative feedback from community.
• VPF sent business impact info.

Challenges - Communication

• Highlight test database more. Which test environment to use to test? Hard to find, confusing.
• Use newer version of Wiki.
• Not everyone got cutover weekend updates. Make it voluntary if interested. Sign-up roster. 
• Expected update for each step - not sure if steps 1-x done.
• Some confusion about UAT start and end dates. Then took 2 days for Lisa to do closing. Delayed due to fixing issue. Show FI close in the schedule from the start. 
• Explore what other testing can continue while close in progress. But have to keep close accurate.
• No idea what is in the enhancement packs. Didn't know it was on the Wiki.
• Send final documentation (validation steps, weekend schedule, ...) earlier.

Schedule - The project schedule was primarily based on prior years experience and it's start and finish times were restricted to a window whose  start is dependent on the time packs are made available by SAP and whose finish be completed before the start of the new Payroll year. Although not entirely flexible, there might be some opportunities to change details within our schedule.

Could combining Systems Integration Testing with User acceptance testing become the rule vs. an option?

Could the development/transport freeze time be reduced provided process owners were available to validate and approve changes outside of the normal business hours? (i.e. nights and/or weekends)

How significant was the impact of bringing down production at noon on the Friday of the cut-over weekend?

                   Project Plan

• • (September - October) - Plan, create master script catalog, improve test scripts, move pending work to production
  • 10/05/2009 - Last date to implement projects until freeze ends on 12/24/2009 (Except Open Enrollment & Union Structures)
  • 10/19/2009 - Production snapshot at end of day - will be used to refresh dev & test environments 
  • 10/22/2009 - Last date to implement support & enhancement changes until freeze ends on 12/24/2009
  • 11/03/2009 - 11/08/2009 Unit Test
  • 11/09/2009 - 11/22/2009 System Integration Test (SIT Task List)
  • 11/23/2009 - 12/09/2009 User Acceptance Test
  • 12/11/2009 - 12/13/2009 Golive Weekend
  • 12/24/2009   Development Freeze ends

Wins - Schedule

• Friday noon well advertised. Worked well. Friday better than Monday.
• Could compress test time, maybe cut days or a week off UAT, but current schedule allows for issues, absences. 
• Freeze period not a problem. Criticals can go during freeze.

Challenges - Schedule

Testing Tools and Materials - Test Cases were located on the Wiki and Request Tracker was utilized for reporting issues. The SAP SH2 environment was updated with the support packs and was refreshed with a snapshot of production data taken on October? 

Were your documented test cases available, current and complete?

Was the test environment adequate for testing all critical business functions?

Was the appropriate data available for processing test scenarios?

 Was Request Tracker an effective tool for logging issues?

Wins - Tools & Materials

• Good to have central repository of all test cases.
• Test environments, recently refreshed good.
• RT worked fine. No issues in reporting problems.
• Liked scenario testing. Do more in the future.

Challenges - Tools & Materials

•  Too many test cases. Go with downloaded list organized by business owner.
• Plan intermediate state testing earlier.

Other (Governance, Process, Project Mgmt, Risk Mgmt, Sponsorship, Staffing Levels, and Teamwork)

Were the correct decision makers involved?

Was there cooperation across different business areas where true integration testing was needed? 

Did the project have the right teams assembled with the appropriate team members?

Wins - Other

•  Gill representing facilities worked.

Challenges - Other

• Steering committee IS&T heavy, business owner light. Gerry O'Toole, Wayne Turner. 
• Or Christine & Gillian are steering committee*.*

Steering Committee - January 20, 2009

At the end of any repeatable software exercise, it is important to review the process to determine ways to make future iterations better.  By better, we mean executing in a shorter period of time, executing with less errors or executing a greater volume. 

For Support Pack 2009, an interview of the teams executing the Support Packs is being conducting in a group, Joint Application Development (JAD) setting.  Challenges or things that negatively impacted individuals are being captured as well as suggestions to remediate these challenges.  This list will be reviewed by the management team to determine steps to be taken to improve the 2010 experience.

Participants should brainstorm everything they felt was successful during the project life-cycle (the "Wins") as well as everything they felt could have been done more efficiently or effectively (the "Challenges") including a recommendation for remediation. This can include but should not be limited to Communications, Governance, Process, Project Mgmt, Risk Mgmt, Sponsorship, Staffing Levels, and Teamwork.

Wins

• Communication
  • No paper handouts were generated during this project. Overhead presentations, emails, system messages and the wiki were effectively utilized.
  •  updates were timely and informative
• Schedule
  • Cut-over down time had no practical impact on Singapore operation.
  • The overall project and each of it's major phases were completed on time inline with the original plan.
  •  contingencies were needed and utilized
• Governance
  •  
• Process
  •  
• Project Management
  •  
• Risk Management
  • Contingency time was and is a must for the cut-over weekend
• Sponsorship
  •  
• Staffing Levels
  •  
• Teamwork
  •  
• Other
  •  

Challenges - Recommendations

• Communication
  •  steering committee mtgs could be scheduled around milestones
• Schedule
  • leave time for pre-development systems preparation
  • SAP delivers the packs for analysis in October
  •  
• Governance
  •  Others are welcome to participate on the steering committee. Strive for balance of SAIS & Business representation.
• Process
  •  
• Project Management
  • Recurring nature of this project could allow for on-the-job training for future managers.  A new role (eg. project coordinator) could be defined to both prepare next year's leader and serve as a back-up as needed.
  •  
• Risk Management
  •  Recommended that we assign back-up managers
• Sponsorship
  •  
• Staffing Levels
  •  
• Teamwork
  •  
•  Other
  •  

Project Extended Team - January 29, 2009

 At the end of any repeatable software exercise, it is important to review the process to determine ways to make future iterations better.  By better, we mean executing in a shorter period of time, executing with less errors or executing a greater volume. 

For Support Pack 2009, an interview of the teams executing the Support Packs is being conducting in a group, Joint Application Development (JAD) setting.  Challenges or things that negatively impacted individuals are being captured as well as suggestions to remediate these challenges.  This list will be reviewed by the management team to determine steps to be taken to improve the 2010 experience.

Participants should brainstorm everything they felt was successful during the project life-cycle (the "Wins") as well as everything they felt could have been done more efficiently or effectively (the "Challenges") including a recommendation for remediation. This can include but should not be limited to Communications, Governance, Process, Project Mgmt, Risk Mgmt, Sponsorship, Staffing Levels, and Teamwork.

Wins

• Communication
  •  Meeting notes maintained on the wiki pages worked well
  • Wiki very helpful for getting updated information
• Schedule
  •  Contingency was used and needed
  •  Dedication of team and willingness to work extra hours
• Governance
  •  
• Process
  •  
• Project Management
  •  Transport Manager role useful
• Risk Management
  •  
• Sponsorship
  •  
• Staffing Levels
  •  
• Teamwork
  •  
• Other
  •  

Challenges - Recommendations

• Communication
  • Meeting notes on the wiki were not always accurate
• Schedule
  • Late-occurring problem with Finance tested contingency 
  • Revise schedule to allow extra time for problems/long-duration steps
  • Schedule for preparation of different systems should reflect available resources on each system
  • Addition of enhancement packs extended time for system preparation more than expected
  • Benchmark enhancement pack application before project starts (if possible)
• Governance
  •  
• Process
  • Late delivery of high-use transactions & programs for analysis/division was difficult & time-consuming
  • Keep identification of critical transactions & reports for reference next year - include reference to test scripts
• Project Management
  • Need individual assigned to monitor issues in RT queue
• Risk Management
  • Need development & testing environments for items that must go into production shortly after go-live
  • Better definition of project freeze and difference between development freeze and transport freeze
• Sponsorship
  •  
• Staffing Levels
  • QA  could play a more predominant role i.e. back-up manager
  • Project manager should be identified earlier and have fewer testing responsibilities
• Teamwork
  •  
•  Other
  •  

Additional input for SAIS team members

Wins

• Including more detail on the Action Log and implementation plan than in the past
• Everyone stepped up to meet the shortened deadlines by working extra hours but this shouldn't become a habit
• Second year of SIT execution
• Now have two years of SIT scripts
• Some business owners / users updated scripts on their own
• Aliases worked well again this year

Challenges

• Implement the support packs and enhancements in a sandbox/prototype environment first to get a better idea of the impact and time needed
• Start review/analysis of the support packs already received earlier - don't perform testing until all are received
• A spreadsheet is not the best tool for keeping the information needed for testing; a DB or other tool would be helpful
• Have an environment with the changes applied available so development can continue
• During the implementation, please don't move status calls earlier without at least an hour's notice
• Extended team meetings were not conducted frequently enough
• Information dissemination was pushed down to the individual functional teams, didn't come from one place
• Many individual test scripts still not updated
• I had responsibility for approving UAT for some business areas due to staff turnover
• Confusion about when UAT ended
• Confusion about what could be tested during period end closing
• Confusion about the intention of SIT / UAT
• Business owners were told they did not have to test during UAT; they could accept the results of SIT as UAT; created a mess that had to be re-communicated at length
• Not enough communication
• Scheduling of users during UAT wasn't done consistently; given a time period and a deadline and told to meet it; created lots of uncertainty around the deadline
• One area that I don't think came out clearly in Friday's meeting was the need to track Transaction Testing (unit test items) and Test Cases (SIT items) in a db as opposed to spreadsheets. The spreadsheet approach forces us into multiple versions, file manipulation and inconsistencies in tracking amongst our groups. If we used a db and initialized it with this past SP testing data our tracking and updating would be much more efficient.
• The other area that needs improvement is Action Log. This log needs to be a little more explicit in what needs to take place in which environment.